Parallel processing in human-machine interface applications

ABSTRACT

A human-machine interface (HMI) application ( 26 ) uses parallel processing. The HMI engineering system ( 24 ) allows explicit specification ( 44 ) of different cores of a multi-core processor ( 16 ) for different elements and/or actions. The programmer may design the HMI application for concurrent operation. The HMI engineering system ( 24 ) or runtime system ( 28 ) may test ( 56 ) for data dependency amongst the elements or actions and automatically assigns different cores where data is independent. During runtime, different threads for the HMI application (e.g., different elements and/or actions) are scheduled for different cores.

RELATED APPLICATIONS

The present patent document claims the benefit of the filing date under35 U.S.C. §119(e) of Provisional U.S. Patent Application Ser. No.61/443,812, filed Feb. 17, 2011, which is hereby incorporated byreference.

BACKGROUND

The present embodiments relate to human-machine interface (HMI)applications. Modern automation systems combine information technologywith industrial machinery to assist the design, implementation, andmonitoring of control systems. A HMI displays machinery data to a humanoperator and receives commands from the operator to control themachinery and process performed by the machinery. HMI devices may beconsidered the “windows” to very complex industrial processes.

HMI applications for operating an HMI device are developed with anengineering system. Engineering systems allow the application developerto create HMI applications without exposing details of the underlyingarchitecture to the application developer. The HMI screen includesdifferent elements placed by the application developer. For each ofthese elements, the application developer creates a list of actions tobe executed each cycle. The list is created from elements withpredefined actions or a list of possible actions for selection. Theseactions may be for changing values (e.g. SetValue) of variables (Tags)or changing an appearance and properties of the elements themselves(e.g. color, X-position, Y-position). An HMI program loop of the HMIapplication is created from the selected elements and correspondingactions.

The engineering system or the runtime process for the HMI devicecompiles the input from the application developer to implement the HMIapplication. The runtime process is performed on the HMI device, whichis implemented with a single-core central processing unit (CPU).

The HMI program loop controls execution of the actions. A single threadtraverses the list of elements of the screen and executes the list ofactions for each element. The scheduling is left to the operatingsystem. The operating system may schedule other threads, such asassociated with communications, into a separate processing unit.However, the execution of the HMI program loop with the elements andactions is typically serialized in a round-robin fashion on everyexecution cycle. This serialization creates a bottleneck that affectsresponse time, rendering, and missed communication deadlines withexternal devices.

SUMMARY

Systems, methods and computer readable media are provided for parallelprocessing in a human-machine interface (HMI) application. The HMIengineering system allows explicit specification of different cores of amulti-core processor for different elements and/or actions. Any numberof cores may be used, such as 2, 4, or 8. The programmer designs the HMIapplication for concurrent operation of HMI elements. Alternatively oradditionally, the HMI engineering system or runtime system tests fordata dependency among the HMI elements or actions and automaticallyassigns different cores where data is independent. During runtime,different threads for the HMI application (e.g., different HMI elementsand/or actions) are scheduled for different cores.

In a first aspect, a system is provided for parallel processing in ahuman-machine interface (HMI) application. At least one sensorconnection and at least one actuator connection for industrial machineryare provided. A multi-core processor electrically connects with the atleast one sensor connector and the at least one actuator connector. Adisplay is operable to display elements associated with the at least onesensor connector and the at least one actuator connector. A memory hasstored the elements for the display and actions corresponding to theelements. The multi-core processor is configured to associate a firstsub-set of the actions with a first core of the multi-core processor,associate a second sub-set of the actions with a second core of themulti-core processor, schedule first and second HMI threads for thefirst and second cores of the multi-core processor based on the firstand second sub-sets, process the first and second HMI threads with thefirst and second cores, and control the at least one actuator and thedisplay of the elements as a function of the processing of the first andsecond HMI threads.

In a second aspect, a method is provided for parallel processing in ahuman-machine interface (HMI) application. HMI elements are establishedfor display on a screen. A plurality of computer processing units of amulti-core processor is listed. An assignment by a user of differentones of the computer processing units to different ones of the HMIelements is received. Actions for each of the HMI elements are set. TheHMI elements, the corresponding actions, and assigned computerprocessing units are stored as an HMI application.

In a third aspect, a non-transitory computer readable storage medium hasstored therein data representing instructions executable by a programmedprocessor for parallel processing in a human-machine interface (HMI)application. The instructions include analyzing dataflow for a pluralityof HMI actions, identifying at least first and second actions with nodata dependency, assigning the first action to a first core of theprocessor, assigning the second action to a second core of theprocessor, implementing first and second threads for the first andsecond actions, and controlling an HMI device based on the implementingof the first and second threads.

Any one or more of the aspects described above may be used alone or incombination. These and other aspects, features and advantages willbecome apparent from the following detailed description of preferredembodiments, which is to be read in connection with the accompanyingdrawings. The present invention is defined by the following claims, andnothing in this section should be taken as a limitation on those claims.Further aspects and advantages of the invention are discussed below inconjunction with the preferred embodiments and may be later claimedindependently or in combination.

BRIEF DESCRIPTION OF THE DRAWINGS

The components and the figures are not necessarily to scale, emphasisinstead being placed upon illustrating the principles of theembodiments. Moreover, in the figures, like reference numerals designatecorresponding parts throughout the different views.

FIG. 1 is a block diagram of one embodiment of a multi-layer system forparallel processing in a human-machine interface (HMI) application;

FIG. 2 is a block diagram of an engineering system for parallelprocessing in a HMI application, according to one embodiment;

FIG. 3 is a block diagram of a runtime system for parallel processing ina HMI application, according to one embodiment;

FIG. 4 is an example display for an HMI application;

FIG. 5 is an example illustration of explicit assignment in anengineering system and runtime scheduling of corresponding threads;

FIG. 6 is a timing diagram comparing examples of sequential and parallelprocessing of an HMI application;

FIG. 7 is a flow chart diagram of one embodiment of a method forparallel processing in an HMI application; and

FIG. 8 is a flow chart diagram of another embodiment of a method forparallel processing in an HMI application.

DETAILED DESCRIPTION OF EMBODIMENTS

There is an increasing demand for more computation capabilities in HMIdevices, such as control panels for industrial machinery. The use ofvideo and imaging in HMI applications is getting more popular. Embeddedprocessors with multiple cores may be used to satisfy these demands.Multi-core CPUs may be superior in terms of concurrency, powerconsumption, and heat dissipation than single-core CPUs. Byparallelizing an HMI application for multi-core systems, betterperformance may be achieved. However, leaving the assignment of themultiple cores to the operating system may not provide optimumimprovement.

HMI applications that make effective use of multi-core processors arecreated. To effectively exploit multi-core processors, the applicationdeveloper and the engineering system provide parallel programmingprimitives to create concurrent HMI applications. The applicationdeveloper has the freedom to explicitly execute parts of the HMIapplication in the processing units of their choice.

The programming primitives for parallelization of the application areused in the engineering system. There are several runtime activitiesthat may be effectively parallelized automatically or without explicituser intervention or assignment. For example, communication to externaldevices, access to the file system to read/write files, and rendering ofuser-interface elements may be parallelized automatically. The actionsof different elements may be parallelized also or instead. Two cases areprovided for parallelization: manual scheduling by the applicationdeveloper, and automatic partitioning and scheduling of specializedactivities by the runtime system. Some actions for elements may beautomatically parallelized without user intervention. This frees theapplication developer who is not familiar with parallel programming fromexplicitly exploiting concurrency. If the application is running on amulti-core system, automatic parallelization makes effective use of thecomputing resources.

Parallel execution of an HMI application may have one or more of thefollowing advantages. Workload for processing the elements in the screenmay be distributed among the available cores and the overall executiontime of the application is reduced. Reduced execution time for triggeredevents may improve the HMI response times and therefore may improve theuser experience when interacting with the application. More elements perscreen may be processed. Current systems limit the amount of elementsper screen to a certain number to be able to provide fast responsetimes. The cycle time of the HMI application may be reduced. Complex andCPU intensive applications, such as high-definition video, may beprocessed more easily with multiple-cores.

FIG. 1 shows one embodiment of a HMI system. The HMI system is a hostcomputer, man-machine interface, and/or graphic user interface forinteraction with and control of programmable logic controllers,actuators, sensors, or combinations thereof.

The HMI system includes a HMI engineering system 24, a HMI application26, and a HMI runtime system 28. Additional, different, or fewercomponents may be provided. Any one or more of the components mayimplement parallelism.

The HMI engineering system 24 is used by the application developer tocreate the HMI application 26. The HMI engineering system is a personalcomputer, workstation, server, or other processor providing apre-defined set of rules for the application developer. For example, thepre-defined rules prohibit the application developer from specifying howthe code in the HMI application 26 is executed in the actual CPUs of theHMI device. The HMI engineering system 24 provides blocks of codeassociated with elements, such as in a visual programming environment.The application developer interacts with the HMI engineering system 24to select elements and corresponding actions for operation of the HMIdevice. The application developer schedules actions, and the HMIengineering system 24 assigns code or the pre-defined rules to thescheduled actions. The selections are compiled, using pre-defined rules,into code for the HMI application. Alternatively, manual programming ofthe code itself may be used. An example HMI engineering system isSIMATIC WinCC Flexible ES, from Siemens Industry.

The HMI application 26 is created for the HMI runtime system 28. The HMIapplication 26 is a program, code, or software for running on the HMIruntime system 28. The HMI application 26 defines the user interface andassociated control of industrial machinery. Sensor inputs, actuatoroutputs, and programmable logic controller instructions, in combinationwith displayed elements, are controlled based on the HMI application 26.Inter-device and programmable logic controller communications or controlfunctions may be provided. The HMI application 26 is designed tointeract with sensors, actuators, controllers, and the physical process.

The HMI runtime system 28 is a HMI device, such as a control panel. TheHMI runtime system 28 executes the HMI application. The HMI application26 is loaded onto and installed in the HMI runtime system 28, such as bydata transfer over a network or from a non-transitory medium (e.g., diskor memory device). SIMATIC WinCC Flexible RT, from Siemens Industry, isan example of a runtime system.

The HMI engineering system 24, the HMI application 26, and/or the HMIruntime system 28 support parallel processing. Parallel programmingprimitives are provided in the HMI engineering system 24 for developmentof the HMI application 26. The primitives are code for selecting,manually, parallelization for different elements and/or actions. Theparallelization primitives are accessible for the application developerto explicitly assign. Alternatively or additionally, automaticparallelization is available for some of the operations and constructsof HMI Applications 26. The HMI engineering system 24 automaticallyassigns elements and/or actions to different computer processing unitsof the same processor. Conflict detection or other dependency analysismay be used for automatic assignment.

The resulting HMI application 26 includes mapping of the elements andactions to threads for specific CPUs. Elements are mapped to threads,and threads to CPUs. Threads are the computation units that encapsulatea set of elements and actions. This may be expressed as mapping theelements or actions to specific CPUs. The HMI runtime system 28 includesthe multi-core processor and operating system for implementing differentthreads in different cores. The threads are assigned by the HMIapplication 26 when running on the HMI runtime system 28 to differentcores. Alternatively, the HMI runtime system 28 may dynamicallyimplement the automated assignment of threads for elements and/oractions to different cores.

FIG. 2 shows one embodiment of a system for parallel processing of a HMIapplication. The system of FIG. 2 represents the HMI engineering system.

The system includes a processor 30, a memory 32, and a display 34.Additional, different, or fewer components may be provided. For example,a user input device is provided.

The computer processing performed by the processor 30 may be implementedin various forms of hardware, software, firmware, special purposeprocessors, or a combination thereof. Some embodiments are implementedin software as a program tangibly embodied on a non-transitory programstorage device. By implementing with a system or program, assignment ofelements and/or actions for parallel processing may be provided as partof creation of the HMI application.

The processor 30 and/or memory 32 are part of a computer, personalcomputer, server, workstation, network processor, or other now known orlater developed processing system. Various peripheral devices such as,for example, the display 34, a disk storage device (e.g., a magnetic oroptical disk storage device), a keyboard, a printing device, and amouse, may be operatively coupled to the processor 30. A program may beuploaded to, and executed by, the processor 30 comprising any suitablearchitecture. Likewise, processing strategies may includemultiprocessing, multitasking, parallel processing and the like. Theprocessor 30 is implemented on a computer platform having hardware, suchas one or more central processing units (CPU), a random access memory(RAM), and input/output (I/O) interface(s). The computer platform alsoincludes an operating system and microinstruction code. The variousprocesses and functions described herein may be either part of themicroinstruction code or part of the program (or combination thereof)which is executed via the operating system. Alternatively, the processor30 is one or more processors in a network.

The instructions, user input, rules, and/or other information are storedin a non-transitory computer readable memory, such as the memory 32. Thememory 32 is an external storage device, RAM, ROM, and/or a local memory(e.g., solid state drive or hard drive). The same or different computerreadable media may be used for the instructions and other data. Thememory 32 may be implemented using a database management system (DBMS)managed by the processor 30 and residing on a memory, such as a harddisk, RAM, or removable media. Alternatively, the memory 32 is internalto the processor 30 (e.g. cache). The memory 32 stores images, elements,actions, HMI device information (e.g., number of cores), sensorconnections, actuator connections, and/or rules.

The instructions for implementing the processes, methods and/ortechniques discussed herein are provided on computer-readable storagemedia or memories, such as a cache, buffer, RAM, removable media, harddrive or other computer readable storage media. Computer readablestorage media include various types of volatile and nonvolatile storagemedia. The functions, acts or tasks illustrated in the figures ordescribed herein are executed in response to one or more sets ofinstructions stored in or on computer readable storage media. Thefunctions, acts or tasks are independent of the particular type ofinstructions set, storage media, processor or processing strategy andmay be performed by software, hardware, integrated circuits, firmware,micro code and the like, operating alone or in combination.

In one embodiment, the instructions are stored on a removable mediadevice for reading by local or remote systems. In other embodiments, theinstructions are stored in a remote location for transfer through acomputer network. In yet other embodiments, the instructions are storedwithin a given computer, CPU, GPU or system. Because some of theconstituent system components and method steps depicted in theaccompanying figures are preferably implemented in software, the actualconnections between the system components (or the process steps) maydiffer depending upon the manner in which the present embodiments areprogrammed.

The display 34 is a CRT, LCD, projector, plasma, printer, or otherdisplay for displaying the options available in the engineering system.For example, the display 34 displays the example screens 70 shown inFIG. 4 or 6. The display 34 assists the application developer inprogramming the HMI application by displaying various elements 72-86 tobe presented to the user of the HMI device.

FIG. 3 shows one embodiment of a system for parallel processing in a HMIapplication. The system of FIG. 3 represents the HMI runtime system,such as a HMI device. One example is a panel for control of industrialmachinery. The system may be a computer, workstation, server, or otherHMI device.

The processor 16, memory 18, and display 20 may be the same or differenttype of device as described for the processor 30, memory 32, and display34 for the system of FIG. 2. The processor 30 of the HMI runtime systemis a multi-core processor. Any number of cores may be provided, such as2-8 cores. Each core is a separate instance of a computer processingunit. The cores are on a same semi-conductor chip, but able to operateindependently and in parallel. Different processes may be performed bythe different cores at the same time.

The multi-core processor 16 electrically connects to the sensor andactuator connectors 12, 14 and any other communications connections. Theelectrical connection is through one or more input or output pins of theprocessor 16, a bus, a communications interface, a semiconductor chip,or other device or devices for facilitating the input and output of theprocessor 16. The electrical connection is on the same circuit board, ina same housing, or is external to the processor 16.

The display 20 of the HMI runtime system is a touch screen in oneembodiment. The touch screen facilitates user interaction with the HMIdevice, such as using soft buttons (e.g., displayed button element 78where the screen detects user contact with the displayed element 78).Alternatively or additionally, dedicated button, knobs, sliders,switches, or other user controls are provided separately from thedisplay.

The display 20 displays the elements associated with one or more sensorsand/or actuators. For example, the screen 70 of FIG. 4 is a screendisplayed for an HMI device. Various elements 72-86 are provided. Oneelement is an image 80. The image 80 may be computer generated or a feedfrom a camera (e.g., a real-time view of the machinery). Another elementis a clock 76. Yet another element is a button 78, presented in thisexample as an emergency stop button. Elements for particular devicesbeing controlled or monitored are provided, such as the reactor 72 andwater tank 74. The device elements may alternatively or additional befor actuators, such as a pump for the water tank 74 and a fuel feedcontrol for the reactor 72. Data elements, such as the temperature 82,84 and pressure 86 are provided. The data elements 82, 84, 86 indicateinformation from or derived from sensors. Other data may be presented,such as programmed information (e.g., a schedule) or historyinformation. Additional, different, or fewer elements may be provided.For example, FIG. 5 shows a screen with additional buttons, an animatedelement, and a progress bar in addition to an image and data displayelements.

The memory 18 of the HMI runtime system stores the elements for displayand actions corresponding to the elements. The actions are implementedas part of the HMI application run by the processor 16. The elements arefor display on the display 20. The elements may have different states,such as red and/or flashing for emergency or error and blue and steadyotherwise. The different possible states and a current state are stored,such as being stored as part of the HMI application (e.g., provided bythe code).

The memory 18 stores a list of cores of the multi-core processor 16. Thelist may be organized in one place or may be distributed, such as beingindicated for different threads.

The memory 18 stores the instructions for the HMI runtime system. Theinstructions include the HMI application. The operating system or rulesfor implementing the HMI application and any components of the HMIdevice may be stored.

The system also includes sensor and actuator connections 12, 14.Additional, different, or fewer connections may be provided. Forexample, additional or fewer sensor or actuator connections areprovided. As another example, one or more connections for communicationswith other HMI runtime systems (e.g., other panels), controllingworkstation or computer, or programmable logic controllers are provided.

The sensor connections 14 are ports of an interface. Physical andelectrical connection is provided for receiving measurements fromsignals. Control information may or may not be provided through theconnection to the sensors. Any number of sensors may be connected to anygiven sensor connection 14, such as in a bussed or sensor addresssystem.

Similarly, the actuator connections 12 are ports of an interface.Physical and electrical connection is provided for transmitting controlsignals to the actuators. Response or measurement information may or maynot be provided through the connection from the actuators. Any number ofactuators may be connected to any given actuator connection 12, such asin a bussed or actuator address system.

The sensors and actuators are for control of industrial machinery. Forexample, the sensors include pressure, temperature, force, position,light, humidity, optical, or other sensors for monitoring an industrialprocess or equipment. As another example, the actuators includepneumatic, hydraulic, electric, or other sources of force for changingthe industrial process or equipment operation. Some example industrialmachinery includes machinery of power facilities, chemical plants,manufacturing facilities, heat and ventilation systems, air conditioningsystems, fire safety systems, reactors, or other collections ofinteracting parts. In other examples, the industrial machinery includesspecific units, such as a cutting machine, a pressing machine, a robot,a tank, a vehicle, or any other device.

Using the HMI engineering system to design the HMI application and/orthe HMI runtime system to implement the HMI application, the multi-coreprocessor 16 of the HMI runtime system is configured to associatedifferent actions with different cores. For example, one sub-set ofactions is configured to be implemented by one core and another sub-setof different actions is configured to be implemented by another core.The different actions may be of a same or different type, but usedifferent data. Parallel programming primitives are provided forconfiguring in the HMI engineering system and the HMI runtime system.The resulting HMI application includes instructions for parallelprocessing.

Other operations than actions associated with the elements may beassigned different cores. For example, communication with sensors and/orprogrammable logic controllers (PLC) is assigned to a particular core.Since the communications may have high priority, the assignment may beto a dedicated core for execution. Alternatively, other operations areassigned to the same core.

In one embodiment, the cores are assigned explicitly by the applicationdeveloper. The application developer assigns the different actions orgroups of actions to different cores. For example, the various actionsare assigned by element. The actions for each given element are assignedto a same core, but the actions for different elements may be assignedto different cores. One or more cores may be assigned actions from aplurality of elements.

The application developer, using an understanding of the architecture ofthe industrial machinery and/or the HMI device, may know which parts ofthe HMI application may be executed in different cores to improveperformance. A learned or natural sense of concurrency in the HMIapplication allows assignment to different cores. The parallelismachieved at the user level may be higher than the parallelism achievedat the runtime or OS level. The HMI engineering system gives applicationdevelopers control over mapping the components of the HMI application tospecific cores.

In the example of FIG. 4, the HMI Application monitors the coretemperature 82 of the nuclear reactor 72, and the temperature 84 andpressure 86 of a water tank 74. The HMI application has an EmergencyStop button 78 that the operator may trigger when the nuclear reactor'score temperature 82 reaches a hazardous level. Additionally, there is aclock 76 showing the time of the day. The active elements of thisapplication may be the temperature 82 of the core, the water tanktemperature 84, the water tank pressure 86, the clock 76, and the button78. The reactor 72 and the water tank 74 may be mere representationsrather than actuators. Alternatively, the reactor 72 and water tank 74are dynamic or may change or are associated with actuators.

The active elements may communicate, such as periodically, with externaldevices (e.g. the programmable logic controller connected to the nuclearreactor 72, the programmable logic controller connected to the watertank 74, and with the temperature and pressure sensors) and communicatelocally to the HMI device (e.g. to get the time of the day). Thisarchitecture has natural concurrency because the reactor temperature 82,the water tank temperature 84 and pressure 86, and the clock 76 arecommunicating with different devices and their execution does not haveany interdependencies. The actions associated with these elements may besafely executed in parallel.

FIG. 5 shows example assigning in the HMI engineering system. A CPUselection is available in addition to the workflow of creating elementsand actions for the HMI application. After or while placing any desiredelements on the screen, the user may program the elements. Theprogramming includes setting an affinity for the element and/or actions.A list of CPU options is provided in a menu for each element. The listincludes all possible cores or a subset of cores (e.g., one or morecores may not be available due to dedication to other processes). Forexample, the CPU selection menu displays all of the CPUs available inthe host computer or the processor of the HMI device/runtime system. Thedeveloper selects an available core from the menu. Alternatively, thedeveloper assigns a number or other value of a variable representing acore with or without user interface presentation of options. Theassignment of the CPU to the element or actions places the element oractions in a thread for the CPU. The thread binds the actions in theelements of the screen to specific CPUs. FIG. 5 shows assignments forthe elements.

In one embodiment, a default option is available to the developer.Rather than selecting between different CPUs, the developer may selector the HMI engineering system automatically assign a “default” settingfor the CPU. For example, the developer may not want to deal with themapping of elements or actions to threads for CPUs, so selects thedefault setting. The default setting may emulate the behavior of asingle-CPU running of the HMI application. The operations for thedifferent elements are assigned to one core and run sequentially by thecore. This may allow backward compatibility with legacy HMI applicationsnot designed to exploit multi-core machines. Alternatively, theselection of the “default” indicates that automated core assignment isto be performed by the HMI engineering system or the HMI runtime system.Instead, a separate “automatic” selection may be provided.

The developer similarly chooses one or more actions from a list ofactions for every element in the screen. The list includes all possibleactions or just possible actions for a given element. Alternatively, thelist includes predetermined actions for a selected element. The CPUselection precedes the selection of the actions, but may be performedafter. Other selections may be performed for programming the elementsand actions, such assigning properties to the elements, settingvariables, and setting values of variables for the selected actions.

In one embodiment, the different actions or elements are automaticallyassigned to the different cores. For example, the developer selects an“automatic” or “default” core in the HMI engineering system.Alternatively, the automatic assignment is performed without userselection or despite user selection. For example, load balancing may beperformed to shift actions from one or more cores to other cores despitedeveloper assignment.

Automatic parallelization allows HMI applications to benefit frommulti-core technology even though the application developer does notexploit concurrency. In some cases, the application developer is notfamiliar with parallel programming or does not have time to think aboutconcurrency and therefore simply leaves all the elements to the“default” setting. In other cases, the HMI application is createdwithout a HMI engineering system that includes core assignmentcapability for the developer. The HMI engineering system or the HMIruntime system may safely parallelize the execution of certain elements.

Concurrency is automatically detected and exploited without theintervention of the developer. Different operations modifying differentdata, and therefore do not have any interdependencies. For example, agraphic element is rendered as an image element on the screen (e.g., seethe animated element or the image element in FIG. 5). The renderinginvolves reading and displaying one type of data (e.g., code for ananimation or a data of a video feed). The operation of the data display(e.g., temperature sensor) may involve a different type of data (e.g.,signals from a temperature sensor). Since different data is used forthese operations, the operations may be safely parallelized by the HMIengineering system and/or the HMI runtime system.

In one embodiment, data dependency is known. Certain elements may nevercreate a data conflict with other elements. For example, the possibleactions that may be associated with a given element use different typesof data than the possible actions that may be associated with adifferent given element. Using this predetermined knowledge, the actionsfor these elements may be automatically assigned to different cores.

In other embodiments, data dependency for a given HMI application istested for automatic assignment. Data dependency is tested with dataflowanalysis. Any dataflow analysis of the actions may be performed, such asdataflow analysis used for compilers. Data dependence may be assumedwhere the testing is inconclusive. The dataflow analysis is performed onthe actions, elements, and data to identify any conflicts. Resourceconflicts may be tested.

Where conflicts are identified, the corresponding actions are to beexecuted sequentially. Where the data or resource usage is independent,the actions and/or elements may be assigned for parallel execution bydifferent cores of the same processor. Different sub-sets of actionsand/or elements are associated with different CPUs based on the dataflowanalysis.

Any assignment priority may be used, such as assigning on a next corebasis. Each data independent element is assigned sequentially to thenext core, with the assignment cycling through the available cores.

In an alternative approach, the assignment is load balanced. Automatedassignment or reassignment from explicitly assigned cores may beperformed based on load balancing. The original explicit assignment mayuse load balancing, such as presenting only cores with lesser loads asselectable options or indicating load information for use in selecting acore.

The processing load for each action or element may be estimated. Theestimate is based on all possible processing or typical processing. Theprocessing may be emulated by the HMI engineering system to determinerelative or estimated processing loads. Alternatively, the processingload may be determined with feedback from actual implementation and usein HMI runtime systems. The processing load may be calculated from anumber of command lines, number of variables, or other programming.

The load is a processing load. The number of operations to be performedby the processor and/or the time to execute is used as the processingload. Other measures may be used. In other embodiments, the number ofcalls to a memory, amount of data loaded, number of commands,communication bandwidth or other criteria is used as the load orincluded in the load determination with the processing load.

Based on the actions or elements, whether previously assigned or beingassigned, the relative load of the different cores is determined. Theactions or elements are assigned to have a more even load for each corewhile maintaining data dependency.

The load balancing may account for priority. For example, somecommunications or operations may be more important, such as actionsassociated with safety. The core executing these operations may receiveless load as part of the load balancing.

FIG. 6 shows an example of assigned and/or runtime parallel processingof actions for different elements as compared to sequential. In theconventional HMI application, the actions of all the elements areprocessed sequentially. None of the actions are parallelized. Otherprocesses, such as setup, drawing, and registering of user actions areprovided. In the multi-core assignment, the processing of the HMIelements is divided between different cores. This division allowssimultaneous or parallel processing, reducing the time needed to processthrough all the elements. As a result, the response time to perform anaction may be reduced.

In another example, the user first clicks a “Start 1” button thattriggers the execution of a script. Before the script finishes, the userclicks a “Start 2” button. This button triggers the execution of adisplay image script that every second reads a picture from memory anddisplays the picture in the HMI Screen. For sequential operation,although “Start 2” is clickable and the event is captured by the system,the user does not experience any image changes until the scriptactivated for “Start 1” execution completes. There is a delay betweenthe triggering event and the actual execution of the image rendering.

Consider an additional event that occurs immediately after the “Start 2”button is pressed and before the script for “Start 1” finishesexecution. This third event is pressing the “Reset” button that triggersa reset for data used by the script for “Start 1.” In this case, a dataconflict exists between the actions triggered by “Reset” and “Start 1”button because the two modify the same data. The “Reset” action must bedelayed until after the script action finishes.

Since the two actions for “Start 1” and “Start 2” modify different data,there are no dependencies. These actions may instead be parallelized.Error! Reference source not found. Performing these actions in parallelmay improve the operator experience. Since the action for “Start 2” isperformed by a different core, there is a more immediate response to theuser's command “Start 2.” The operator experiences the performance ofthe script and the image display simultaneously.

The HMI engineering system separates the actions into different threads.The operating system of the HMI runtime system executes the threads inthe assigned cores. The different threads for the actions and/orelements are scheduled for operation in the assigned cores of themulti-core processor. Different sub-sets of actions are performed bydifferent cores.

FIG. 5 shows one example processing of the different threads for thedifferent cores. The multi-core processor is configured by the threadsof the HMI application or creation of the threads by the multi-coreprocessor itself. A main-loop thread is run in one core, such as adefault or primary CPU. The main-loop thread integrates the execution ofthe different cores. The main-loop thread calls the threads for theother cores, including the threads for the elements of the HMIapplication. The HMI runtime main program loop dispatches the workloadamong the CPUs. A worker thread is provided for each CPU and for anydefault setting that executes the actions of the elements attached tothat CPU. The CPU executing the main-loop program may also execute oneor more threads for elements. For example, FIG. 6 shows CPU0 runningboth the main-loop program and processing of one or more elements.Alternatively, the CPU running the main-loop program does not alsoexecute threads for elements.

The main-loop program synchronizes the threads. After the worker threadsfinish their jobs (i.e., complete execution). Synchronization by themain program loop guarantees correctness of the HMI application and datadependencies. The synchronization avoids data dependency by timingrepetitive execution until after the threads of the various elements arecomplete. The synchronization may avoid data irregularity or cross callsfor the same data. Alternatively, threads for the elements may repeatwithout synchronization.

By running the HMI application on the HMI device with a multi-coreprocessor, the industrial machinery may be controlled. One or moreactuators, display of the elements representing the HMI, communications,control data, sensor readings, data processing, or other operations arecontrolled based on the processing of the different threads. Forexample, a visual characteristic (e.g., color, size, blinking or not, orposition) of a displayed element changes based on execution of a threadfor the element. As another example, an actuator or programmable logiccontroller is instructed to take action based on execution of a threadfor the corresponding element. The actions for the different elementsare performed in parallel by processing the threads. Some actions may beperformed sequentially.

FIG. 7 shows a method for parallel processing in a human-machineinterface (HMI) application. The method is implemented by the systems ofFIG. 1, FIG. 2, and/or FIG. 3. The method is provided in the ordershown, but other orders may be provided. Additional, different or feweracts may be provided. For example, acts 50 and/or 52 are not performed.Acts 40, 42, 44, 46, and 48 may be performed as part of a HMIengineering system for creating the HMI application. As another example,acts 50 and 52 are performed without acts 40-48. Acts 50 and 52 areperformed in a HMI runtime system for implementing the HMI application.

In act 40, HMI elements are established for display on a screen. Adeveloper selects one or more elements from a library of the HMIelements. In a visual programming environment, any desired HMI elementis cut-and-pasted, dragged, or otherwise placed on a screen. Code forimplementation of the elements is associated with the visualrepresentation. In other embodiments, the elements are programmed. Theelements appropriate for a given HMI application are selected and/orcreated.

In act 42, the computer processing units (CPUs) are listed. The CPUs ofthe processor that is to run the HMI application being developed arelisted. The list may be displayed in a menu. The menu may allowselection of a given CPU by the developer for binding the CPU to anelement and associated actions. The list may be displayed separatelyfrom the assignment of act 44, such as listing in a separate image or aprinted list. In other embodiments, the list is not collected ordisplayed in a single form or location, but is a list in the sense ofbeing known.

Based on the list of available CPUs, the developer inputs an indicationof the CPU for each element or action. For example, the developer placesa new element on a screen. The element is added. Upon placement or in aseparate workflow, the developer is provided a menu, input field, orother location to indicate a CPU to be associated with the element. Forexample, the developer is prompted to select a CPU from a displayed listof available CPUs for the element. Alternatively, the developer programsthe element to be assigned to the CPU. The process is repeated for eachelement. Rather than element level input, the developer may input theaffinity for each action.

In act 44, the HMI engineering system receives an assignment of theelement or action to a CPU. The signals from the input are received andprocessed to determine the binding. By repeating the input andcorresponding reception of the selection, different ones of the CPUs areassigned to different HMI elements and/or actions.

The received assignments are from the developer. In an additional oralternative embodiment, the received assignments are from an automatedassignment. For example, the developer fails to select or input anassignment, inputs a “default” or automatic assignment, or both. Someassignments may be explicit while others are not (left to be automated).

Whether always automated or in response to an indication of applicationof automated assignment, the HMI engineering system automaticallyassigns the HMI elements and/or actions to different CPUs. Datadependency is reviewed for conflicts. Actions or elements that may beexecuted separately are assigned to the same or different CPUs.

Any assignment criteria may be used for automated binding. For example,load balancing is performed. The threads for the HMI elements assignedto a default computer processing unit or not assigned are distributedbetween the different ones of the computer processing units. Thedistribution is performed to result in similar loading across the CPUs.

In response to the reception, the HMI elements and/or actions are boundto specific CPUs using automated or explicit processes. The selectedCPUs for the different elements are assigned to execute the actionsand/or elements.

In act 46, the actions are set for the HMI elements. The developerprograms the HMI application further. For each element, one or moreactions are selected or assigned. The actions control the operation ofthe element and corresponding communication for that operation. Theactions provide for receipt of information, display of information,and/or transmit of control signals or data.

Values for variables, display or visual characteristics, and/or otherinformation may be selected or assigned. For example, the x-position,y-position, read and write I/O functions from sensors and actuators,manipulation of files, setting and monitoring alarms, or other settingsare configured. The developer completes programming of the HMIapplication. Once complete, the programming indicates different actionsto be executed by different CPUs.

The HMI application is compiled or otherwise completed from theprogramming. The various assignments and selections are formed intoinstructions executable by the multi-core processor. The threads forimplementing the HMI application are created by the HMI engineeringsystem. Alternatively, the compiling and/or creation of the threads areperformed by the HMI runtime system.

In act 48, the HMI application is stored. The HMI elements,corresponding actions, and assigned CPUs are stored as the HMIapplication. The complete HMI application is saved for use in one ormore HMI devices. The HMI application is stored locally at the HMIengineering system. Alternatively or additionally, the HMI applicationis transferred for storage elsewhere, such as being transferred over anetwork to HMI devices. The storage is to any type of memory, such as acache, a hard drive, solid state, flash, removable, optical, ormagnetic.

In act 50, the HMI application is loaded on an HMI device. The HMIdevice is a runtime system for implementing the HMI application. The HMIdevice includes the multi-core processor to perform the actions for theHMI application. The display screen, multi-core processor,communications connectors, and other components used by the HMIapplication to control or monitor the industrial machinery are providedas part of the HMI device. By loading the HMI application, the HMIdevice may be configured to implement the HMI application.

The HMI device may install the HMI application. The installation mayinclude further processing, such as automated assignment of threadsbetween different CPUs and creation of a main-program loop thread.

In act 52, the HMI application is run by the HMI device. Upon power upor activation of the HMI application, the multi-core processor loads andprocesses the threads for the HMI application. The actions for theelements of the HMI application are implemented by the CPUs based on theassignments from the user or created automatically.

FIG. 8 shows a method for parallel processing in a HMI application. Themethod is for automatic assignment of CPUs to elements or actions of theHMI application. The method is implemented by the systems of FIG. 1,FIG. 2, and/or FIG. 3. The method is provided in the order shown, butother orders may be provided. Additional, different or fewer acts may beprovided. For example, acts 62 and/or 64 are not performed. Acts 54, 56,58, and 60 may be performed as part of a HMI engineering system forcreating the HMI application or as part of an HMI runtime system forimplementing the HMI application. As another example, acts 62 and 64 areperformed without acts 54-60. Acts 62 and 64 are performed in a HMIruntime system for implementing the HMI application.

In act 54, default assignment of actions or elements is received. Thedefault may be a designation as “default” or may be a lack of anydesignation. At least some or all of the actions and/or elements are notexplicitly assigned to different CPUs. In alternative embodiments, allof the actions and/or elements are assigned, but may be reassigned.

In response to receipt of the default or unassigned setting for one ormore actions and/or elements, an analysis may be automatically begun.The receipt is of a trigger event, such as loading the HMI application,addition of an element, compiling the HMI application, or selection of a“default” setting.

In act 56, the dataflow for the HMI actions is analyzed. The datadependency for an action is determined as the action is added or afterall actions have been added. The analysis is eventually of all theactions for HMI elements displayed on the screen of the HMI device.

By comparing the data and/or processes used by each action, anyconflicts may be identified. Groups of actions associated with differentelements that use the same resources are implemented sequentially in asame core. The dataflow analysis identifies the different groups. Thegroups of actions are tested for conflicts in concurrent execution.

Groups of actions associated with different elements that have noconflict may be implemented in parallel. In act 58, the actions withoutdata or other resource dependency are identified. Concurrent executionwill not result in resource conflicts.

In act 60, the actions are assigned to different cores. Differentactions are assigned to different cores. The assignment may be by group,such as assigning by elements. The actions of the same element areassigned to the same core, but actions of different elements may beassigned to different cores. One or more elements may be assigned toeach core. The assignment is based on there being a lack of datadependency for the actions of the elements.

The assignment may include other criteria. For example, load balancingis considered. The grouping of actions and/or elements by core isassigned to provide similar execution burdens on the cores. Thebalancing may consider priority and/or operational differences for thedifferent cores.

In act 62, the threads for the different assignments are implemented bythe different cores. A main-loop thread calls the threads for thevarious elements and/or cores. The cores execute the thread for theelements assigned to the respective cores. The threads may besynchronized by a main-loop or primary core process.

In act 64, a HMI device is controlled. By implementing the threads inthe different cores, the associated actions and elements are providedfor interaction with the industrial machinery and/or an operator. Basedon scheduled actions, operator overrides, operator input, sensor input,or other data, the HMI application controls the HMI device and therelated industrial machinery. The display of various elements mayrespond to sensed data, action implementation, operator input, actuatoractivation, or other information based on the HMI application.

Implementing a HMI application programmed for sequential operation(e.g., single core) in a multi-core processor may provide someimprovement as compared to implementation in a single-core processor.For two cores compared to one core execution, an improvement in responsetime may be about 14% given operating system-based assignment of the HMIapplication to one core and assignment of other component processes toanother core. The theoretical improvement for two cores is 50% or 15.35ms. Similarly, four cores may improve the response time by about 21%when the theoretical limit is 75% or 7.675 ms. Eight cores may improvethe application by about 22% when the theoretical limit is of 88% or3.83 ms. Relying on mere operating system assignment results in lowparallel efficiency. Using explicit or automated assignment of actionsor elements of an HMI application may result in efficiency closer to thetheoretical. By introducing parallel programming primitives, theefficiency gap from theoretical may be reduced.

Various improvements described herein may be used together orseparately. Although illustrative embodiments of the present inventionhave been described herein with reference to the accompanying drawings,it is to be understood that the invention is not limited to thoseprecise embodiments, and that various other changes and modificationsmay be affected therein by one skilled in the art without departing fromthe scope or spirit of the invention.

What is claimed is:
 1. A system for parallel processing in ahuman-machine interface (HMI) application, the system comprising: atleast one sensor (14) connection for industrial machinery; at least oneactuator (12) connection for the industrial machinery; a multi-coreprocessor (16) electrically connected with the at least one sensor (14)connection and the at least one actuator (12) connection; a displayoperable to display elements associated with the at least one sensor(14) connection and the at least one actuator (12) connection; a memory(18) having stored the elements for the display and actionscorresponding to the elements; wherein the multi-core processor (16) isconfigured to associate a first sub-set of the actions with a first coreof the multi-core processor (16), associate a second sub-set of theactions with a second core of the multi-core processor (16), schedulefirst and second HMI threads for the first and second cores of themulti-core processor (16) based on the first and second sub-sets,process the first and second HMI threads with the first and secondcores, and control the at least one actuator (12) and the display of theelements as a function of the processing of the first and second HMIthreads.
 2. The system of claim 1 wherein the multi-core processor (16)is part of an HMI runtime system.
 3. The system of claim 1 wherein themulti-core processor (16) is configured to associate the first andsecond sub-sets of the actions based on a user assignment.
 4. The systemof claim 1 wherein the multi-core processor (16) is configured toperform a dataflow analysis of the actions and associate first andsecond sub-sets based on the dataflow analysis.
 5. The system of claim 4wherein the multi-core processor (16) is configured to determine datadependency from the dataflow analysis and associate based on the datadependency.
 6. The system of claim 1 wherein the multi-core processor(16) is part of an HMI device (28).
 7. The system of claim 1 wherein theelements comprise a button, a progress bar, an image, a data display, orcombinations thereof, and wherein the multi-core processor (16) isconfigured to control at least one of the elements by altering a visualcharacteristic.
 8. The system of claim 1 wherein the multi-coreprocessor (16) is configured to run a main-loop thread in a third core,the main-loop thread calling the first and second HMI threads.
 9. Thesystem of claim 8 wherein the multi-core processor (16) is configured tosynchronize the first and second HMI threads as part of the main-loopthread.
 10. The system of claim 1 wherein the multi-core processor (16)is configured to load balance across the first and second cores as afunction of execution time, the associations of the first and secondsub-sets of actions being a function of the load balance.
 11. A methodfor parallel processing in a human-machine interface (HMI) application,the method comprising: establishing (40) HMI elements for display on ascreen; listing (42) a plurality of computer processing units of amulti-core processor (16); receiving (44), from a user, an assignment ofdifferent ones of the computer processing units to different ones of theHMI elements; setting (46) actions for each of the HMI elements; andstoring (48) the HMI elements, the corresponding actions, and assignedcomputer processing units as an HMI application.
 12. The method of claim11 wherein establishing (40), listing (42), receiving (44), setting(46), and storing (48) are performed with an engineering system forcreating the HMI.
 13. The method of claim 11 further comprising: loading(50) the HMI application on an HMI device (28), the HMI device (28)comprising the multi-core processor (16); and running (52) the HMIapplication with the actions implemented by the computer processingunits according to the assignments from the user.
 14. The method ofclaim 11 wherein establishing (40) the HMI elements comprises providinga library of the HMI elements with lists of actions, and wherein setting(46) the actions comprises selecting, by the user, the actionscorresponding to the HMI elements.
 15. The method of claim 11 whereinlisting (42) comprises displaying a menu for selection of the computerprocessing units for one or more of the HMI elements.
 16. The method ofclaim 11 wherein receiving (44) comprises receiving (44) an assignmentfor a plurality of the HMI elements to a default one of the computerprocessing units; further comprising: load balancing threads for the HMIelements assigned to the default computer processing unit between thedifferent ones of the computer processing units.
 17. The method of claim11 wherein receiving (44) comprises binding the HMI elements andcorresponding actions to specific ones of the computer processing unitsbased on user selection of the specific ones.
 18. In a non-transitorycomputer readable storage medium having stored therein data representinginstructions executable by a programmed processor (30) for parallelprocessing in a human-machine interface (HMI) application, the storagemedium comprising instructions for: analyzing (56) dataflow for aplurality of HMI actions; identifying (58) at least first and secondactions with no data dependency; assigning (60) the first action to afirst core of the processor (16); assigning (60) the second action to asecond core of the processor (16); implementing (62) first and secondthreads for the first and second actions; and controlling (64) an HMIdevice (28) based on the implementing of the first and second threads.19. The non-transitory computer readable storage medium of claim 18wherein the HMI actions are associated with HMI elements displayed on ascreen of the HMI device (28), wherein analyzing and identifyingcomprise analyzing and identifying groups of the actions associated withdifferent HMI elements, and wherein assigning comprises assigning basedon the no data dependency being for the groups of actions for thedifferent HMI elements.
 20. The non-transitory computer readable storagemedium of claim 18 wherein analyzing dataflow comprises testing forconflicts in concurrent execution, and wherein identifying comprisesidentifying the at least first and second actions with no datadependency as having no conflicts in concurrent execution.
 21. Thenon-transitory computer readable storage medium of claim 18 whereinimplementing comprises calling the first and second threads from amain-loop thread implemented on a third core and synchronizing the firstand second threads with the main-loop thread.
 22. The non-transitorycomputer readable storage medium of claim 18 further comprisingbeginning, automatically, the analyzing in response to a default coresetting (46) for the first and second actions, and wherein assigning thefirst and second actions comprises load balancing between the first andsecond cores.